Control of transport dynamics in overlay networks

Transport control is an important factor in the performance of Internet protocols, particularly in the next generation network applications involving computational steering, interactive visualization, instrument control, and transfer of large data sets. The widely deployed Transport Control Protocol is inadequate for these tasks due to its performance drawbacks. The purpose of this dissertation is to conduct a rigorous analytical study on the design and performance of transport protocols, and systematically develop a new class of protocols to overcome the limitations of current methods. Various sources of randomness exist in network performance measurements due to the stochastic nature of network traffic. We propose a new class of transport protocols that explicitly accounts for the randomness based on dynamic stochastic approximation methods. These protocols use congestion window and idle time to dynamically control the source rate to achieve transport objectives. We conduct statistical analyses to determine the main effects of these two control parameters and their interaction effects. The application of stochastic approximation methods enables us to show the analytical stability of the transport protocols and avoid pre-selecting the flow and congestion control parameters. These new protocols are successfully applied to transport control for both goodput stabilization and maximization. The experimental results show the superior performance compared to current methods particularly for Internet applications. To effectively deploy these protocols over the Internet, we develop an overlay network, which resides at the application level to provide data transmission service using User Datagram Protocol. The overlay network, together with the new protocols based on User Datagram Protocol, provides an effective environment for implementing transport control using application-level modules. We also study problems in overlay networks such as path bandwidth estimation and multiple quickest path computation. In wireless networks, most packet losses are caused by physical signal losses and do not necessarily indicate network congestion. Furthermore, the physical link connectivity in ad-hoc networks deployed in unstructured areas is unpredictable. We develop the Connectivity-Through-Time protocols that exploit the node movements to deliver data under dynamic connectivity. We integrate this protocol into overlay networks and present experimental results using network to support a team of mobile robots

System Design and Algorithmic Development for Computational Steering in Distributed Environments

Supporting visualization pipelines over wide-area networks is critical to enabling large-scale scientific applications that require visual feedback to interactively steer online computations. We propose a remote computational steering system that employs analytical models to estimate the cost of computing and communication components and optimizes the overall system performance in distributed environments with heterogeneous resources. We formulate and categorize the visualization pipeline configuration problems for maximum frame rate into three classes according to the constraints on node reuse or resource sharing, namely no, contiguous, and arbitrary reuse. We prove all three problems to be NP-complete and present heuristic approaches based on a dynamic programming strategy. The superior performance of the proposed solution is demonstrated with extensive simulation results in comparison with existing algorithms and is further evidenced by experimental results collected on a prototype implementation deployed over the Internet

Optimizing Network Performance of Computing Pipelines in Distributed Environments

Supporting high performance computing pipelines over wide-area networks is critical to enabling large-scale distributed scientific applications that require fast responses for interactive operations or smooth flows for data streaming. We construct analytical cost models for computing modules, network nodes, and communication links to estimate the computing times on nodes and the data transport times over connections. Based on these time estimates, we present the Efficient Linear Pipeline Configuration method based on dynamic programming that partitions the pipeline modules into groups and strategically maps them onto a set of selected computing nodes in a network to achieve minimum end-to-end delay or maximum frame rate. We implemented this method and evaluated its effectiveness with experiments on a large set of simulated application pipelines and computing networks. The experimental results show that the proposed method outperforms the Streamline and Greedy algorithms. These results, together with polynomial computational complexity, make our method a potential scalable solution for large practical deployments

Transcription network construction for large-scale microarray datasets using a high-performance computing approach

BACKGROUND: The advance in high-throughput genomic technologies including microarrays has demonstrated the potential of generating a tremendous amount of gene expression data for the entire genome. Deciphering transcriptional networks that convey information on intracluster correlations and intercluster connections of genes is a crucial analysis task in the post-sequence era. Most of the existing analysis methods for genome-wide gene expression profiles consist of several steps that often require human involvement based on experiential knowledge that is generally difficult to acquire and formalize. Moreover, large-scale datasets typically incur prohibitively expensive computation overhead and thus result in a long experiment-analysis research cycle. RESULTS: We propose a parallel computation-based random matrix theory approach to analyze the cross correlations of gene expression data in an entirely automatic and objective manner to eliminate the ambiguities and subjectivity inherent to human decisions. We apply the proposed approach to the publicly available human liver cancer data and yeast cycle data, and generate transcriptional networks that illustrate interacting functional modules. The experimental results conform accurately to those published in previous literatures. CONCLUSIONS: The correlations calculated from experimental measurements typically contain both “genuine” and “random” components. In the proposed approach, we remove the “random” component by testing the statistics of the eigenvalues of the correlation matrix against a “null hypothesis” — a truly random correlation matrix obtained from mutually uncorrelated expression data series. Our investigation into the components of deviating eigenvectors after varimax orthogonal rotation reveals distinct functional modules. The utilization of high performance computing resources including ScaLAPACK package, supercomputer and Linux PC cluster in our implementations and experiments significantly reduces the amount of computation time that is otherwise needed on a single workstation. More importantly, the large distributed shared memory and parallel computing power allow us to process genomic datasets of enormous sizes

CHEETAH: Circuit-Switched High-Speed End-to-End Transport Architecture Testbed

We propose a circuit-switched high-speed end-to-end transport architecture (CHEETAH) as a networking solution to provide high-speed end-to-end circuit connectivity to end hosts on a dynamic call-by-call basis. Not only is it envisioned as a complementary service to the basic connectionless service provided by today’s Internet; it also relies on and leverages the presence of this service. Noting the dominance of Ethernet in LANs and SONET/SDH in WANs, CHEETAH circuits will consist of Ethernet segments at the ends and Ethernet-over-SONET segments in the wide area. In this article we explain the CHEETAH concept and describe a wide-area experimental network testbed we have deployed based on this concept. The network testbed currently extends between Raleigh, North Carolina, Atlanta, Georgia, and Oak Ridge, Tennessee, and uses off-the-shelf switches. We have created CHEETAH software to run on end hosts to enable automated use of this network by applications. Our first users of this network testbed and software will be the Terascale Supernova Initiative (TSI) project researchers, who plan to use this network for large file transfers and remote visualizations

Fusion of threshold rules for target detection in wireless sensor networks

We consider a network of sensors distributed in a target area providing environmental measurements that are subject to normally distributed, independent additive noise. Each sensor node applies a threshold rule to the measurements to decide the presence of a target; the distance to the target together with the threshold value determines its hit and false alarm probabilities or rates using a signal attenuation model. We propose a centralized threshold-OR fusion rule for combining the individual sensor node decisions. Under the statistical independence of sensor measurements, we derive fusion threshold bounds using Chebyshev’s inequality based on individual hit and false alarm probabilities but without requiring a priori knowledge of the underlying probability distributions. We derive conditions to ensure that the fused method achieves a higher hit rate and lower false alarm rate compared to the weighted averages of individual sensor parameters. The simulations using Monte Carlo method illustrate significant detection performance improvements of the proposed fusion approach

Transcription Network Construction for Large-Scale Microarray Datasets using a High-Performance Computing Approach

Background: The advance in high-throughput genomic technologies including microarrays has demonstrated the potential of generating a tremendous amount of gene expression data for the entire genome. Deciphering transcriptional networks that convey information on intracluster correlations and intercluster connections of genes is a crucial analysis task in the post-sequence era. Most of the existing analysis methods for genome-wide gene expression profiles consist of several steps that often require human involvement based on experiential knowledge that is generally difficult to acquire and formalize. Moreover, large-scale datasets typically incur prohibitively expensive computation overhead and thus result in a long experiment-analysis research cycle. Results: We propose a parallel computation-based random matrix theory approach to analyze the cross correlations of gene expression data in an entirely automatic and objective manner to eliminate the ambiguities and subjectivity inherent to human decisions. We apply the proposed approach to the publicly available human liver cancer data and yeast cycle data, and generate transcriptional networks that illustrate interacting functional modules. The experimental results conform accurately to those published in previous literatures. Conclusions: The correlations calculated from experimental measurements typically contain both genuine and random components. In the proposed approach, we remove the random component by testing the statistics of the eigenvalues of the correlation matrix against a null hypothesis - a truly random correlation matrix obtained from mutually uncorrelated expression data series. Our investigation into the components of deviating eigenvectors after varimax orthogonal rotation reveals distinct functional modules. The utilization of high performance computing resources including ScaLAPACK package, supercomputer and Linux PC cluster in our implementations and experiments significantly reduces the amount of computation time that is otherwise needed on a single workstation. More importantly, the large distributed shared memory and parallel computing power allow us to process genomic datasets of enormous sizes

Design of scalable parallel algorithms for large-scale data analysis and visualization

https://scholarlycommons.pacific.edu/soecs-facbooks/1002/thumbnail.jp